Scientists develop AI-enabled post-production tools for independent Canadian filmmakers
A Simon Fraser University lab is developing AI-enabled post-production tools to extend the creative potential of professional filmmakers, with the aim of positioning the Canadian movie industry as an even stronger player in the global entertainment space.
SFU's Computational Photography Lab, which began four years ago with the hiring of Yağız Aksoy, has been focusing on solving fundamental computer vision problems, while setting up a research studio to conduct computational photography research in an active production environment.
"Our aim is to bring the capabilities of CGI tools such as the Unreal Engine to post-production software for live-action scenes," says Aksoy, a professor in SFU's School of Computing Science, whose research in computer vision seeks to understand light and geometry from photographs and videos.
"With a physical understanding of real scenes, CGI capabilities such as controlling light sources and the motion of cameras become possible for real-world footage."
His recently opened research studio, located on SFU's Burnaby campus, features a dozen or so cameras arrayed at different elevations and angles before a green screen. Currently, Aksoy and his team produce educational videos and then use those same videos in their research. Aksoy says his goal is to invite local independent filmmakers and film students to produce movies with them.
"This will help us understand the everyday challenges and the expectations that the creatives have, and also allow creatives to tell AI what to do, which will guide our research direction."
Ultimately, his work builds towards next-generation post-production tools with a specific focus on supporting low-budget, and independent Canadian movie productions. These AI-enabled tools have the potential to change what we can expect from a typical post-production pipeline. With the physical control of the scene given to creative professionals, he says he hopes to see the visual language of light as a storytelling device get richer and a deeper field of specialization.
He is careful to delineate that the work of the Computational Photography Lab is analytical AI, differentiating it from the generative AI products like ChatGPT or Stable Diffusion. Analytical AI methods analyze an image to get the light and geometry so that a filmmaker can then change it. It is the creative making the decision as to what the product should look like, not a computer.
"The way that we make movies and the way we tell stories have been evolving together with the technology," says Aksoy, pointing to the new potential story-telling devices enabled by his research, including a recent paper about developing an AI approach to intrinsic decomposition that works on a wide range of images.
"When you make profits the focus, AI becomes a threat to replace creative jobs," he stresses, "but AI is just a tool."
He hopes his creative-driven approach to AI research will enrich the visual language of storytelling, while allowing Canadian independent filmmakers to produce better movies with smaller budgets.
Provided by Simon Fraser University